Symmetric positive-definite conformation-tensors are ubiquitous in models of viscoelasticity. In this paper, the multiplicative decomposition of the conformation tensor is revisited. The nonuniqueness in this decomposition is exploited (i) to ensure stationarity of the decomposed dynamics whenever the conformation tensor is stationary, and (ii) to impose gauge conditions (cf. symmetric square root, or Cholesky decomposition) in the dynamics, for both deterministic and stochastic settings. The general procedure developed in this paper is exemplified on the upper-convected Maxwell model, and a (typically) increased numerical accuracy of the modified dynamics is found